Borders cannot be removed if a wall is on it.
As title says, if you need/want to remove a border you have to remove everything on it, some things make sense, however walls/pillars do not because they can be placed on a external of a border. The dismantle function should check if walls/pillars on it share space with a different border that would allow them to exist. Effectively only stopping you if it's the last border you remove.
That, or it could even make it turn into a ruin wall.