I think of brutalism generally as distinct from modernism, although early brutalism still seems to have a moderne/primitive aspect about it. I have mixed feelings about brutalist architecture, but my negative opinion of it is colored in part by Boston's disastrous West End Redevelopment Project of the 1960s, which destroyed an entire neighborhood in the name of "urban renewal." The neighborhood was leveled and replaced by high-rise condos and two massive, imposing, austere brutalist government structures. One of these has appeared in a number of films. It was designed by Paul Rudolph, who earlier had built a number of amazing modern homes in Sarasota, FL.
Was never there, but based on photos, I would agree.