Although the Supreme Court’s opinion stated that a pardon carries “an imputation of guilt and acceptance of a confession of it,” this was part of the Court’s dictum for the case. Whether the acceptance of a pardon constitutes an admission of guilt by the recipient is disputed. In Lorance v. Commandant, USDB (2021) the 10th Circuit Court of Appeals ruled that "there is no confession and Lorance does not otherwise lose his right to petition for habeas corpus relief for his court-martial conviction and sentence.

That’s from the Wikipedia article about the case you’re referring to. Note that a dictum from the Supreme Court is influential but not binding as a precedent.

However, that case has to do with a preemptive pardon for a crime that the recipient had not yet been accused of. Pardons for a crime that the recipient is currently being punished for may be different. (If the government wants to remove you from prison, can you refuse to leave?) So may pardons for a crime for which the recipient has already completed his punishment.