Exodus 16:29, 30 See, for that the LORD has given you the Sabbath, therefore He gives you on the sixth day the bread of two days; you abide every man in his place, let no man go out of his place o the seventh day. So the people rested on the seventh day.
That's the original purpose of the Sabbath. If you leave your place to go somewhere you are breaking the Sabbath. When the temple was built people were actually breaking the Sabbath by going to another place on that day. That's another reason why God established the Lord's Day under the New Covenant, He wants us to restore Saturday as a day of true rest and physical restoration.