Active Surveillance and Chemotherapy Before Surgery in Treating Participants With Stage II-III Rectal Cancer

Inclusion Criteria: * Histologically confirmed diagnosis of rectal adenocarcinoma * Eligible for curative resection of rectal adenocarcinoma * Rectal tumor location =\< 12 cm from the anal verge as determined by endoscopy or magnetic resonance imaging (MRI) (if endoscopy report is not available or deemed inadequate my treating oncologist) * Nodal involvement confined to the radiation field * Radiologically measurable or clinically evaluable disease as defined in the protocol * Eastern Cooperative Oncology Group (ECOG) performance status (PS): 0, 1 or 2 * Clinical Stage: Stage II and III. N2 disease is to be estimated as four or more lymph nodes that are \>= 10 mm. Clinical staging should be estimated based on the combination of the following assessments: physical exam by the primary surgeon including digital rectal exam (DRE), computed tomography (CT) or positron emission tomography (PET)/CT scan of the chest/abdomen/pelvis and a pelvic MRI. If a pelvic MRI is performed, it is acceptable to perform CT of the chest/abdomen, omitting CT imaging of the pelvis. PET/CT is optional. * No known contraindication to standard (fluoropyrimidine-based) pelvic chemoradiation (e.g. dihydropyrimidine dehydrogenase \[DPD\] deficiency) * Patient of child-bearing potential is willing to employ adequate contraception during treatment and after treatment, as directed by treating clinical team * Willing to provide written informed consent * Willing to return to enrolling medical site for all study assessments Exclusion Criteria: * Diagnosis of inflammatory bowel disease (IBD) * Diagnosis of MSI-H colorectal cancer at time of consent * Recurrent rectal cancer * Tumor is causing symptomatic bowel obstruction (patients who have diverting ostomy are eligible) * Any prior pelvic radiation * Other invasive malignancy undergoing active treatment. Patients receiving prior treatment that precludes standard chemoradiation or ability to receive consolidation/adjuvant chemotherapy will be excluded from survival analyses * Patients unwilling or unable to undergo pelvic MRI