The neuroimaging community is steering towards increasingly large sample sizes, often highly heterogeneous as they are acquired by multi-site consortia. Pooling data across such heterogenous sources requires careful inspection, consistency, and compliance of the acquisition protocols i.e., ensuring different sites and scanners have used the identical or compatible MR physics parameter values. Maintaining accurate protocols and harmonizing them across scanners and sites has been an ad-hoc and error-prone process, often manually managed and insufficiently monitored. Assessing the protocol compliance of a dataset under question to a pre-defined protocol is a necessary step. However, this is often overlooked for lack of realization that parameter values are often modified and/or improvised. Ensuring compliance is an arduous and error-prone process for many reasons including difficulties in working with complicated DICOM standard. To address this issue, we developed an assistive tool that can interface with multiple dataset formats (DICOM, BIDS, XNAT) and automatically generate protocol compliance report. In this talk, we describe the origins of the issue, and demonstrate its presence and importance based on our analysis of over 20 large open neuroimaging datasets on OpenNeuro. We found that many of these datasets have serious issues in acquisition parameters deviating from predefined values, that can potentially render downstream analyses invalid. We propose a practical QA solution to prevent the issue as well as offer open source tools to achieve and monitor protocol compliance in large neuroimaging datasets.