What happens if the shares are not bought in the auction?

If the shares could not be bought in the auction i.e. if shares are not offered for sale in the auction, the transactions are closed out as per SEBI guidelines.

The guidelines stipulate that "the close out Price will be the highest price recorded in that scrip on the exchange in the settlement in which the concerned contract was entered into and up to the date of auction/close out OR 20% above the official closing price on the exchange on the day on which auction offers are called for (and in the event of there being no such closing price on that day, then the official closing price on the immediately preceding trading day on which there was an official closing price), whichever is higher.

Since, in the rolling settlement the auction and the close out takes place during trading hours, the reference price in the rolling settlement for close out procedures would be taken as the previous day's closing price.