Is this normal? They are building cells (and laying in them) from the top of the lower box frames to the bottom of the upper box frames. If I try to lift an upper frame the cells on the bottom, attached to the lower frames are destroyed.
Yes this is normal, it is called burr comb. For some reason the "bee space" has been compromised just a little. Anything over 3/8" and they will build comb in and less than 3/8" they will plug with propolis. Not much to do but scrape it out of the way and carry on, or figure out why you have too big of a gap between the bottom and top frames and correct it. If you are using plastic or even wax foundation, in the spring it is a good place for them to raise drone brood since the cells are larger.
It is normal and since it is quite typically laid up in drone brood it is an excellent place to do spot checks for varroa mites. The varroa shows up real nicely on white pupae.
Old beekeepers thought at least one task of being an adequate beekeeper (vs a bee haver) was to scrape the burr comb off anytime you did a full inspection. Some folks would even go to the trouble of caring around a little pail, catching the burr comb and rending what remained after the girls had cleaned up the mess into wax. One of my old mentors did this religiously one summer (he was perhaps 12 or so) and with the gathering bought himself his first bicycle.