If you do it carefully, it will not affect your rankings
Google knows this https://webmasters.googleblog.com/.../https-as-ranking...
If you do it right, you will get good results.
If you don't have the resources for it, we recommend you wait until you're fully prepared.
If you do it wrong, you run the risk of having your site corrupted and you will lose rankings.
Also, the previous backlinks will not be affected if you configure your access to forward from http version to https version.