I also think the people should be able to overrule the court via referendum

The court arguably has the responsibility of protecting minorities from the majority-elected government. Allowing the public to vote on court rulings, especially in combination with Congress doing a jury-style selection of justices, would just further politicize the court and erode minority rights.