The new option is available in your AdSense control panel and allows you to run square, slightly rounded and very rounded corners. You can see examples of the new ad layout here.
Access the different options through the ‘Setup Tab’ of AdSense but keep in mind that if you have a blended approach to creating ads (your borders being the same color as the background of the ad and your blog) that having square or rounded corners doesn’t matter in the slightest as you don’t see them.
While it’s nice that Google is giving more options to publishers, this one falls into the almost useless category. The best performing Google ads for me has always been the ones without any corners. I do not run a border around my Google ads. This helps the ad blend better with the content and that increases the click rate. However, I can understand that there are some publishers who feel that ads and content should be completely separated and the rounded corner gives them another display option.