First, we agree that God is all-powerful, and does whatever He wishes (Psalm 115:3). Human beings are not in charge, the Lord is king. Second, we agree that God is executing an eternal plan of His own devising (Ephesians 1:11). God learns nothing new, unlike how we learn. Third, we agree that whatever God chooses to do, it is always righteous. God is light, and in Him is no darkness at all (1 john 1:5). Fourth, we agree that God doesn't directly cause everything. He upholds all things (Hebrews 1:3). God causes no one to sin (James 1:13). He controls the circumstances in which temptations come, He already foreknows how we will respond to those temptations, and He chooses to permit us to be tempted and fall, but He Himself doesn't cause anyone to sin.
The word "predestine" appears six times in the New Testament. The Romans, the Jews, Pontius Pilate, and Herod did that which God beforehand predestined them to do (Acts 4:27-28). God predestined those whom He foreknew to be conformed to the image of His son (Romans 8:29). Those whom He predestined He then called, justified, and glorified (Romans 8:30). God predestined that the hidden wisdom which He later revealed to the apostles (which we have in the form of the New Testament) would be to our glory (1 Cor. 2:7). God predestined us to be adopted as His sons through Christ (Eph. 1:5). We inherited an eternal inheritance through Christ, and to this we were predestined (Eph. 1:11).
What truths do we learn from these six short references?
1. Predestination includes sin. God ordained the murderers of Jesus to kill Him, exactly when, where, and how they did. They did it for their own reasons, out of their own hearts, but God used them (unknowingly to themselves) for His own higher purpose.
God does not just predestine the good actions of good people, but also predestines the bad actions of of wicked people, so that His plan comes to pass. This reminds us of Genesis 50, where Joseph said that his brothers selling him into slavery, which they did freely out of their own bad hearts, was at the same time a manifestation of God's higher plan.
2. God is King and Judge of the universe at the same time. This means He, as King, ordains many human actions administratively that He, as Judge, hates morally. This duality of roles is unique to God. I can't think of any human role which comes near to it.
3. God from eternity foreknew specific people. God knows everything because He is omniscient, but God did not foreknow everyone in the Romans 8:29 sense. Everyone will not be conformed to the image of Christ. This tells us that "foreknow" has a much narrower meaning than simply God being cognizant of something. Jeremiah 1:3 says that God "knowing" an individual in this way is parallel to God consecrating that person to something. God "knew" Jeremiah while he was still in the womb, and appointed him to be a prophet.
4. Those whom God foreknew will infallibly be resurrected unto glory (conformed to the image of His Son). Christ is the first-born from among the dead, so He is the first-born of many brothers yet to be resurrected. They are predestined to that outcome, which means their wills are not the final determining factor. This is why a true Christian can never be lost, not even if he (insanely) should wish to be. Eternal security is based on the eternal predestination of God.
5. God only calls (converts), justifies, and glorifies the predestined ones. He doesn't call, justify, or glorify anyone else. God calls people with the Gospel outwardly, to their ears, but Paul was talking about the Holy Spirit's converting call to the heart.
6. God decided from eternity past that He would keep some of His wisdom hidden until the New Testament era, and only then reveal it to Paul and the other apostles. The apostles' wills, virtue, and piety had nothing to do with when God revealed His hidden wisdom. God gave Paul special knowledge about the Church, for instance, that was unknown in the Old Testament era (see Ephesians 3).
7. God predestined certain specific people to become adopted as His sons through Christ. This is both our soul-status of adoption (1 John 3:1) and the as-yet-future experience of our bodies being raised to glory ((Romans 8:23). The word "adoption" refers to both blessings. The status of adoption is ours now, through faith, and resurrection in glory is our future inheritance.
There are two more redemptive facts about predestination which need to be kept in mind.
One is that, if God did not predestine to eternal life, everyone without exception would hate Him into a condemned eternity. The human race is evil. We hate God by our very nature. We don't change into sinners somewhere along the way. We are born in sin, in the womb. We come forth from the womb speaking lies. Predestination is God intervening in individual lives, so as to bring about conversion and salvation.
Second, God has not predestined everyone. Even though He makes it clear to the world that He is willing that they should come to Him, and that He would receive them if they did come, they do not. So the world's rejection of God logically precedes predestination, in the God/man relationship. God didn't from eternity foresee innocent people and predestine them to be evil. He foresaw evil people, and intervened to predestine some of them to come to Him.