Muswellbrook Shire Council has lost an appeal against a Land and Environment Court decision upholding the department's assessment of rehabilitation at Mount Arthur, but won an appeal that leaves the department to pay for its own legal costs.

The council took legal action against BHP company and Mount Arthur mine owner Hunter Valley Energy Coal in 2017 after alleging the mine was not complying with a 2014 Planning Assessment Commission condition of consent on rehabilitation. The council argued the condition required superior "natural landform" rehabilitation over the entire mine site, and not just areas mined in future.

The Department of Planning joined the legal case after approving a mine rehabilitation strategy in 2017 the council argued was inconsistent with the Planning Assessment Commission condition of consent.

In April Land and Environment Court Justice John Robson ruled against the council after finding the Department of Planning's acceptance of an "aspirational" rehabilitation strategy was legally reasonable, in part because mining and rehabilitation are "dynamic and ongoing".

In his decision Justice Robson accepted the council harboured a "genuine concern" that rehabilitation was not being conducted appropriately, but found there was no indication the Department of Planning had "embraced" the "more exacting conditions" supported by the council.

In a decision today the NSW Court of Appeal rejected the council's appeal against Justice Robson's decision, but criticised the Department of Planning for a "shoddily prepared" 2014 project modification approval document after the mine's expansion until 2026 was approved.

The court also questioned the department's delegation of the final rehabilitation strategy to a department employee who approved the strategy "without identifying the instrument by which he did so".

The appeal court said the then Department of Planning secretary Carolyn McNally "played an active role" in the court proceedings, "generally in support of the company's submissions".

The court ordered the department to pay its own legal costs after finding there was "no reasonable basis to doubt that Muswellbrook Shire Council brought the proceedings in the public interest" and out of concern for the future visual impact of the giant mine site after coal mining ends.

In a statement Muswellbrook Council said it respected the Court of Appeal decision but "the community has been woefully let down today by the NSW Department of Planning and the Planning Assessment Commission as a result of their shodding drafting of an important document".

"It is right and proper that the Court of Appeal should admonish the department for taking an active role in the litigation in support of BHP and impose cost consequences as a result of the department's decision to do so," the council said.

"It is important to note that the NSW Government denies councils the ability to apply for merit reviews of mining outcomes they consider to have undesirable impacts on their communities. Councils are limited, therefore, to administrative appeals.

"The NSW Government, however, allows mining companies the benefit of full merit appeals against decisions about which they are not happy. As a result, councils continue to fight for their communities with one hand tied behind their backs."

The council noted Justice Robson "was at pains to point out the decision might have been different had he been considering the question on merit".

A spokesperson for BHP said the company welcomed the decision.

"We take our obligation to rehabilitate disturbed land very seriously, and will continue to work in partnership with the community and other stakeholders in our rehabilitation designs," the spokesperson said.

"At Mount Arthur Coal rehabilitation has commenced on all land that has been made available for remediation, with over 1200 hectares of final rehabilitation already achieved. In addition, we have also undertaken aerial seeding of areas that are not available for final rehabilitation. Following community feedback we have introduced and now use leading practice landform design in rehabilitation at the mine, to create a more natural looking landform shape."

Rehabilitation was inspected by the NSW Government each year and rehabilitation outcomes were reported by the company in its annual environmental management report.