Depending on your theme, they may be hard coded into a sidebar or footer file. If this is the case, you will need to get your hands dirty and edit your theme files.
But you can usually remove them by adding a widget, as these items often only show up when no widgets have been added. So go to Appearance -> Widgets and add the widgets that you want in the appropriate sidebar.